Image processing system with led flicker mitigation

ABSTRACT

The disclosure relates to technology for processing images to detect LED on pulses within the images. An image processing device and method includes a fusion processor and process configured to receive the sequential long and short exposure images and output a fusion image including added corresponding regions of adjacent sequential long and short exposure images. The image processing device and method also includes an LED detection processor and method configured to receive the fusion image, a WDR image and the sequential long and short exposure images and output a control signal indicating whether each corresponding region includes an LED flicker or motion. The image processing device and method also includes a blending processor and method responsive to the control signal to generate a blended image for the sequential long and short exposure images.

FIELD

The disclosure generally relates to image processing, and specificallyan image processing pipeline which reduced LED flicker artifacts in animage stream.

BACKGROUND

In current imaging systems flickering lighting and objects with changingillumination in an image may result in missing parts of an object, orobject distortion. In particular, light-emitting diode (LED) trafficsigns (which can flicker several hundred times per second), and LED stopand head lights of modern cars, have been widely adopted for trafficcontrol signs and vehicle components. However, these LEDs presentdetection issues for current image processing systems. Typically, imagesensors acquire light asynchronously relative to the scenery beingcaptured. This means that portions of an image may not be exposed forpart of the frame duration. This is especially true for objects withillumination flickering when integration times are much shorter than theflicker periods. Zones in an image that are not fully exposed to dynamicscenery may result in object distortion, saturation data loss, and colorartifacts.

Vehicle cameras are required to be able to capture images with a widedynamic range (WDR) of light conditions, such as entering or exitingtunnels. Most of these cameras are equipped with a CMOS sensor with WDRtechnology. The concept of this WDR sensor is to capture an image bycombining multiple exposure frames, with each exposure frame havingdifferent exposure duration. A WDR module composites images by selectingthe short-exposure frames for the areas of movement, as well asoversaturated areas, within an input image, while the dark areas, aswell as non-moving areas (without brightness oversaturation) aresynthesized by the long-exposure frames.

Short exposure times will sometimes cause the image sensor to miss theLED “on” pulse and then cause the image to have flickering artifacts, asshown in the following figure. This flickering issue not only affectsthe viewing experience but also may degrade the accuracy of judgement ina system like Advanced Driver Assistance System (ADAS).

BRIEF SUMMARY

According to one aspect of the present disclosure, there is provided animage processing device, including a fusion processor configured toreceive sequential long and short exposure images, and to generate afusion image including added corresponding portions of adjacentsequential long and short exposure images. The image processing devicealso includes an LED detection processor configured to receive thefusion image, the sequential long and short exposure images, and a widedynamic range (WDR) image created from the sequential long and shortexposure images, and to generate a control signal indicating whether thecorresponding region includes an LED flicker or motion. The imageprocessing device also includes a blending processor responsive to thecontrol signal to generate a blended image.

Optionally, in any of the preceding aspects, the image processing devicemay be configured such that the fusion processor is configured to outputthe fusion image by adding pixel values from the adjacent long and shortsequential images together. Optionally, in any of the preceding aspects,the image processing device may include a fusion processor adding pixelvalues on a pixel-by-pixel basis in corresponding regions of theadjacent long and short sequential images. Optionally, in any of thepreceding aspects, the image processing device may include a fusionprocessor which includes a saturation checking mechanism operating on apixel-by-pixel basis. Optionally, in any of the preceding aspects, theimage processing device may include a blending processor which selects apixel value for a blended image from a corresponding pixel value in thefusion image, or in the WDR image, or by blending of the fusion and WDRimages based on the control signal. Optionally, in any of the precedingaspects, the image processing device may include an LED detectionprocessor configured to determine whether at least a correspondingregion of two adjacent sequential long and short exposure images, thefusion image, and the WDR image includes motion or an LED flicker basedon the relationship between pixel values in the adjacent sequential longand short images. Optionally, in any of the preceding aspects, theblended image is either from the WDR image, from the fusion image, orfrom the blending of the WDR and the fusion images.

One general aspect includes the image processing device where theblended image includes an image showing an LED light source during along exposure image from an image capture device which captures andoutputs the sequential long and short exposure images.

According to one other aspect of the present disclosure, there isprovided a computer-implemented method for processing images, including:receiving sequential long and short exposure images and a wide dynamicrange (WDR) image based on adjacent sequential long and short exposureimages. The computer-implemented method also includes generating afusion image based on the adjacent sequential long and short exposureimages, this fusion image including added data from each of the adjacentlong and short exposure images. The computer-implemented method alsoincludes generating a control signal based on the fusion image, the WDRimage, and the adjacent sequential long and short exposure images, thiscontrol signal indicating whether pixels in the fusion image or the WDRimage include an LED flicker. The computer-implemented method alsoincludes generating a blended image responsive to the control signal.

Optionally, in any of the preceding aspects, the computer-implementedmethod includes generating a fusion image that includes adding togetherpixel values from the adjacent long and short sequential images.Optionally, in any of the preceding aspects, the computer-implementedmethod includes generating a fusion image generated by adding pixelvalues on a pixel-by-pixel basis in corresponding regions of theadjacent long and short sequential images. Optionally, in any of thepreceding aspects, the computer-implemented method includes generating afusion image that includes limiting pixel saturation on a pixel-by-pixelbasis. Optionally, in any of the preceding aspects, thecomputer-implemented method includes generating a blended image thatincludes selecting a pixel value for the blended image from one of acorresponding pixel value in the fusion image, or from the WDR image, orfrom both of a corresponding pixel value in the fusion image and in theWDR image based on the control signal. Optionally, in any of thepreceding aspects, the computer-implemented method includes generating acontrol signal determining whether at least a corresponding region oftwo adjacent sequential long and short exposure images, the fusionimage, and the WDR image includes motion or an LED flicker. Optionally,in any of the preceding aspects, the computer-implemented methodincludes generating a blended image that includes determining whether aregion in the WDR image, the fusion image or the blended image isoversaturated and correcting the oversaturated region.

According to still one other aspect of the present disclosure, there isprovided a non-transitory computer-readable medium storing computerinstructions for processing images, that when executed by one or moreprocessors, causes the one or more processors to perform the steps of:receiving sequential long and short exposure images from an imagesensor; generating a wide dynamic range (WDR) image based on adjacentsequential long and short exposure images; generating a fusion imagebased on the adjacent sequential long and short exposure images, thefusion image including added data from each of the adjacent long andshort exposure images, and generating a control signal based on thefusion image, the WDR image and the adjacent sequential long and shortexposure images, the control signal indicating whether pixels in thefusion image or the WDR image include an LED flicker. The non-transitorycomputer-readable medium also includes generating a blended imageresponsive to the control signal by selecting data from one of the WDRimage, or the fusion image, or the blended combination.

Optionally, in any of the preceding aspects, the non-transitorycomputer-readable medium includes generating a fusion image thatincludes adding together pixel values from the adjacent long and shortsequential images on a pixel-by-pixel basis in corresponding regions ofthe adjacent long and short sequential images. Optionally, in any of thepreceding aspects, the non-transitory computer-readable medium includesgenerating a fusion image that includes limiting pixel saturation on apixel-by-pixel basis. Optionally, in any of the preceding aspects, thenon-transitory computer-readable medium includes generating a controlsignal determines whether at least a corresponding region of twoadjacent sequential long and short exposure images, the fusion image,and the WDR image includes motion or an LED flicker. Optionally, in anyof the preceding aspects, the non-transitory computer-readable mediumincludes generating a blended image includes determining whether aregion in the WDR image, the fusion image or the blended image isoversaturated and correcting the oversaturated region.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter. The claimed subject matter is not limited to implementationsthat solve any or all disadvantages noted in the Background.

BRIEF DESCRIPTION OF THE DRAWINGS

Aspects of the present disclosure are illustrated by way of example andare not limited by the accompanying figures for which like referencesindicate elements.

FIG. 1 illustrates an LED sign in which all LED components are visible.

FIG. 2 illustrates the LED sign of FIG. 1 in which all LED componentsare not visible as a result of faulty imaging.

FIG. 3 is a timing diagram illustrating the output timing of a multipleexposure image sensor.

FIG. 4 is a block diagram of an image processing system.

FIG. 5 illustrates a block diagram of a network system that can be usedto implement various embodiments.

FIG. 6 is a flow diagram illustrating the functions of a WDR unit usedin the image processing system.

FIG. 7 is a flow diagram illustrating the functions of a fusion unitused in the image processing system.

FIG. 8 is a flow diagram illustrating the function of an LED detectionunit.

FIG. 9 is a flow diagram illustrating the function of a blending unitused in the image processing system.

FIG. 10 is a flow diagram illustrating the process performed by ablending processor or blending module in accordance with the technology.

DETAILED DESCRIPTION

The present disclosure will now be described with reference to thefigures, which in general relate to a novel image processing systemwhich allows conventional CMOS image detectors to be utilized to detectenvironmental LED lighting while preventing the introduction ofdistortions and motion artifacts in the final produced image. The imageprocessing system includes a wide dynamic range (WDR) processor ormodule, a fusion processor or module, a LED detection processor ormodule, and a blending processor or module. The system is configured toreceive sequential long and short exposure images from an image sensor.The system uses a WDR image from the WDR processor and a fusion imagefrom the fusion processor in conjunction with the sequential long andshort exposure images to detect LED pulses having a different pulseduration from the image sensor producing the long and short exposureimages. The LED detection processor outputs a control signal indicatingwhether the corresponding region in sequential long and short exposureimages includes an LED flicker or motion, and the blending processorgenerates a blended image for the sequential long and short exposureimages with a flicker-free final image.

It is understood that the present embodiments of the disclosure may beimplemented in many different forms and that claims scopes should not beconstrued as being limited to the embodiments set forth herein. Rather,these embodiments are provided so that this disclosure will be thoroughand complete and will fully convey the inventive embodiment concepts tothose skilled in the art. Indeed, the disclosure is intended to coveralternatives, modifications and equivalents of these embodiments, whichare included within the scope and spirit of the disclosure as defined bythe appended claims. Furthermore, in the following detailed descriptionof the present embodiments of the disclosure, numerous specific detailsare set forth in order to provide a thorough understanding. However, itwill be clear to those of ordinary skill in the art that the presentembodiments of the disclosure may be practiced without such specificdetails.

FIG. 1 illustrates an LED sign 100 showing the word “OPEN” wherein thesign is made up of a plurality of LED lights. As illustrated in FIG. 3,each LED making up the word OPEN has a series of ON pulses recurring at10 ms intervals. The ON pulses are undetectable to the human eye, butcan be detected by imaging systems. As Illustrated in FIG. 3, widedynamic range (WDR) systems using CMOS sensors capture an image bycombining multiple exposure frames, with each exposure frame havingdifferent exposure duration. FIG. 3 illustrates the long and shortsequential exposure frame timing at 220. In the context of thisdisclosure, the term “frame” is used synonymously with the term “image”as a frame may comprise one of many still images which compose thecomplete moving picture.

As a result of the exposure timing between long and short exposureframes, an imaging system may miss the exposure pulse of an LED sign,resulting in the image appearing as illustrated in FIG. 2: a partial orfully, underexposed image.

As illustrated in FIG. 3, the timing disparity between an image capturesensor adapted to output sequential long and short exposure images andan LED “on” pulse is illustrated. Timing graph 210 illustratessequential, periodic pulses of an LED. In one embodiment, the LED onpulse repeats every 10 ms, but the period may be longer or shorterdepending on the LED manufacturer and the use of the LED. An imagecapture sensor captures long exposures (242, 244, 246) and shortexposures (232, 234, 236) in sequence. During a first capture sequence250 comprising long exposure 242 and short exposure 232, the shortexposure 232 will capture partial exposure data of a corresponding LEDpulse 213, while the long exposure 242 will capture data from pulse 212and 213. During a second capture sequence 252 comprising long exposure244 and short exposure 234, the short exposure 234 will capture noexposure data of a LED pulses 214 or 215, while the long exposure 244will capture a full exposure of data from pulse 214. During a thirdcapture sequence 254 comprising long exposure 246 and short exposure244, the short exposure 236 will again only capture a partial exposuredata of a corresponding LED pulse 216, while the long exposure 244 willcapture a full exposure of data from pulse 215.

The detected image of a conventional WDR sensor is composited from theshort-exposure frames in areas of movement, as well as oversaturatedareas. On the other hand, the dark areas, as well as non-moving areas(without brightness oversaturation), an image may be synthesized by thelong-exposure frames. This results in the sign 100 appearing asillustrated in FIG. 2.

FIG. 4 illustrates an image signal processor (ISP) in accordance withthe present technology. The ISP 400 is coupled to a multiple exposuresensor 410 and includes a fusion processor 420, a WDR processor 430, anLED pulse detection processor 440 and a blending processor 450. Themultiple exposure sensor 410 provides sequential long and short exposureimages or frames. The exposure data output from the multiple exposuresensor 410 is provided to the fusion processor 420, WDR processor 430and the LED pulse detection processor 440.

The fusion processor 420 adds multiple exposure data from sequentiallong and short images together. The addition of multiple exposure datatogether is performed on a pixel-by-pixel basis over correspondingregions of pixels in adjacent long and short images. In one aspect, aregion may be an area of pixels in a frame. Regions may be defined by anumber of pixels in width by a number of pixels in height. In oneembodiment, a region may comprise an area comprising a 30 pixels wide by5 pixels high area. Pixels are arranged within a frame in a series ofrows and columns, and hence a region may have the same correspondingregions in adjacent frames (next sequential long or short frame)positioned at the same row and column area in the adjacent frames. Inone embodiment, the fusion processor 420 may include a saturationchecking mechanism to limit final output pixel value to avoidoversaturation in bright exposure regions. In an alternative embodiment,no oversaturation mechanism is used in the fusion processor andoversaturation processing may occur in the blending processor 450.

The output of the fusion processor 420 comprises a fusion image which isprovided to the LED pulse detection processor 440 and the blendingprocessor 450.

The WDR processor 430 outputs a wide dynamic rage image (WDR image). TheWDR image is a clear, no motion-blur picture. In order to output the WDRimage, the WDR processor 430 is configured to detect potential motion bycomparing long exposure frames to short exposure frames. Therefore, itmay recognize the on-off pulse of the LED as motion. When this occurs,the WDR processor 430 outputs the short-exposure frame and, as discussedabove with respect to FIG. 3, the short-exposure frames often miss the“on” pulse of LED light. The WDR image is output to the LED pulsedetection processor 440 and the blending processor 450. In oneembodiment, the WDR processor 420 may include a saturation checkingmechanism to avoid oversaturation in bright exposure regions of the WDRimage. In an alternative embodiment, no oversaturation mechanism is usedin the WDR processor 430 and oversaturation processing may occur in theblending processor 450.

The LED pulse detection processor 440 determines whether a region withininput images is in an LED flickering area or not. Because areas ofmovement can appear similar to LED flickering, the LED detectordetermines whether a raw exposure input includes a true moving area oran LED flickering. If that determination is incorrect, it will causemotion blur or motion discontinuity artifacts in an output image. TheLED pulse detection processor 440 uses the raw exposure data from themultiple exposure sensor 410, the fusion image from the fusion processor420 and the WDR image from the WDR processor 430 to make thisdetermination as to whether a region is an LED pulse or motion. Asdescribed below, the LED pulse detection processor 440 outputs a controlsignal to the blending processor 450 which instructs the blendingprocessor 450 how to process the region in a final blended image.

The blending processor 450 creates and outputs a blended image,comprising a synthesis of the fusion image and the WDR image. Theblended image provides is the blended result of a WDR image and thefusion image according to the information from the LED pulse detectionprocessor 440. The blended image is a more accurate representation ofthe area imaged when the area includes an LED source. As noted above, inembodiments where the WDR processor 430 and fusion processor 420 do notinclude an oversaturation mechanism controlling oversaturation in theirrespective output images, the blending processor 450 may include anoversaturation mechanism.

Each of the processors illustrated in FIG. 4 may comprise circuitryspecifically constructed to perform the functions described herein. Eachprocessor may likewise comprise any suitable programmable hardwareelement such as programmable processing circuitry. Each processor may beprovided as a separate circuit element to other processors or theprocessors may share some or all of their processing circuitry. Each ofthe aforementioned processors may be coupled by one or more systembusses functionally illustrated by the arrows linking the data betweenthe various processors. The arrows indicating data flow are not to beconsidered indicative of the number of placement of data or controlbusses between the processors but merely illustrative of data flowbetween the processors.

FIG. 5 is a block diagram of a network device 500 that can be used toimplement various embodiments. Specific network devices may utilize allof the components shown, or only a subset of the components, and levelsof integration may vary from device to device. Furthermore, the networkdevice 500 may contain multiple instances of a component, such asmultiple processing units, processors, memories, transmitters,receivers, etc. The network device 500 may include a central processingunit (CPU) 510, a memory 520, a mass storage device 530, and an I/Ointerface 560 connected to a bus 570. The bus 570 may be one or more ofany type of several bus architectures including a memory bus or memorycontroller, a peripheral bus or the like.

A multiple exposure sensor 590 is coupled to bus 570 and may beequivalent to multiple image sensor 410 discussed with respect to FIG.4. Sensor 590 is coupled to bus 570 and outputs sequential long andshort exposure images to memory 520 which may be operated on by the CPU.

The CPU 510 may comprise any type of electronic data processor. Thememory 520 may comprise any type of system memory such as static randomaccess memory (SRAM), dynamic random access memory (DRAM), synchronousDRAM (SDRAM), read-only memory (ROM), a combination thereof, or thelike. In an embodiment, the memory 520 may include ROM for use atboot-up, and DRAM for program and data storage for use while executingprograms. In embodiments, the memory 520 is non-transitory. In oneembodiment, the memory 520 includes a fusion module 520A which maycomprise instructions to the CPU 510 to add multiple exposure data fromsequential long and short pulses together and, optionally, instructionsto implement a saturation checking mechanism to limit final output pixelvalues in the fusion image to avoid oversaturation in bright exposureregions. The memory 520 may further include a WDR module 520B whichincludes instructions to the CPU 510 to create and output a WDR image ina manner equivalent to the WDR processor 430. The memory 520 may furtherinclude an LED pulse detection module 520C comprising instructions tothe CPU 510 to determine whether a corresponding region within inputfusion images, WDR images and sequential long and short exposures fromthe exposure sensor 590 is in an LED flickering area or not, operatingin a manner equivalent to the LED pulse detection processor 440. Thememory 520 a blending module 520D includes instructions to the CPU 510to create and outputs a blended image, comprising a synthesis of thefusion image and the WDR image, and provides a more accuraterepresentation of whether an area or region within an image is an LED ornot. The blending module 520D operates in a manner equivalent to theblending processor 450. The blending module 520D may optionally includeinstructions to implement a saturation checking mechanism to limit finaloutput pixel values in the blended image to avoid oversaturation inbright exposure regions where such saturation checking is optionally notincluded in the WDR module 520B and/or the fusion module 520A.

The mass storage device 530 may comprise any type of storage deviceconfigured to store data, programs, and other information and to makethe data, programs, and other information accessible via the bus 570.The mass storage device 530 may comprise, for example, one or more of asolid state drive, hard disk drive, a magnetic disk drive, an opticaldisk drive, or the like.

The network device 500 also includes one or more network interfaces 550,which may comprise wired links, such as an Ethernet cable or the like,and/or wireless links to access nodes or one or more networks 580. Thenetwork interface 550 allows the network device 500 to communicate withremote units via the networks 580. For example, the network interface550 may provide wireless communication via one or moretransmitters/transmit antennas and one or more receivers/receiveantennas. In an embodiment, the network device 500 is coupled to alocal-area network or a wide-area network 580 for data processing andcommunications with remote devices, such as other processing units, theInternet, remote storage facilities, or the like.

FIG. 6 is a flowchart illustrating the processes performed by the WDRprocessor or the WDR module 520B in accordance with the technology. At600, the image sensor 410 or 590 captures multiple sequential long andshort images. It should be understood that the image sensor capture isnot a part of the WDR processor and that the image sensor 410 or 590 maybe any of a number of conventional CMOS technology based image sensorswhich are configured capture and output a plurality of sequential longand short exposure images.

At 610, the WDR processor 430 or WDR module 520B receives a stream ofsequential long and short exposure images. At 620, the long exposureframe is compared to the short exposure frame. This comparison may occuron a region by region basis and for corresponding regions in sequentialframes or multiple sequential (long and short) frames. At 630, adetermination is made as to whether the image (or the region) includesmotion. The determination may be the result of comparing exposure datain adjacent images and within regions to analyze whether similarexposure data in close pixels within a region appear in successiveimages. If so, then motion may be determined. If motion is determined at630, then a blended WDR image comprising the long exposure and shortexposure image is output as the WDR image at 650 based on the comparisonat 630. If motion is not determined to be in the image, then a longexposure image as the WDR image is output at 640.

FIG. 7 is a flowchart illustrating the process performed by the fusionprocessor 420 (or fusion module 520A). At 700, the fusion processor 420or fusion module 520A receives a stream of sequential long and shortexposure images. It should be understood that the time sequence of astream of sequential long and short exposure images will be time-syncedwith the stream received by the WDR processor (or WDR module 430) andthe LED detection processor 440 (or LED detection module 520C) so thatall processors (or modules) operate on the same time-sequenced image(s)within the stream.

At 710, for each common region within two successive images and at 720for each image within each common region, the steps at 730-770 arerepeated to generate a fusion image. At 730, common regions in adjacentsequential short and long exposure images are aligned and determined. Asnoted, the common regions may be the same region relative to the imagerow and column positions in successive images or multiple successiveimages. The process may be performed two adjacent images or multiplesuccessive images. At 740, pixel data in the short exposure image(s) isadded to pixel data in the adjacent sequential long exposure image(s).If, at 750, the added data is over an intensity threshold, a saturationchecking mechanism is used to apply color correction to an output image.If the added data in successive frames, when added, would be over anintensity threshold at 750, then color correction is applied at 760. Thethreshold may be set at any suitable level by a process designed so thatoversaturation of the pixel/region does not occur. Color correction at760 may comprise applying a multiplier factor to the intensity data toreduce the intensity of the data coming from one or more of the longexposure image or short exposure image which will contribute to thefusion image. At 770, the added pixel data (or color corrected data) areoutput and the process moved so the next pixel and next region until allregions in the images are processed. Once all regions are processed at780, the whole fusion image is generated.

FIG. 8 is a flowchart illustrating the process performed by the LEDpulse detection processor 440 or LED pulse detection module 520C. At800, the LED pulse detection processor 440 or LED pulse detection module520C receives a stream of sequential long and short exposure images.Again, the stream of sequential long and short exposure images will betime-synced with the fusion image and the WDR image when received by theLED pulse detection processor 440 or LED pulse detection module 520C.

At 810, for each common region within two successive images and at 820for each image within each common region, the steps at 830-870 arerepeated to generate a control signal to the blending processor 450 orblending module 520D which instructs the blending processor 450 orblending module 520D on how to process a blended image. At 830, commonregions in adjacent sequential short and long exposure images arealigned and determined as in step 730.

At 840, pixel data in the short exposure image(s), adjacent sequentiallong exposure image(s), the fusion image and the WDR image are compared.At 850, based on data in each corresponding region and each pixel, adetermination is made as to whether the pixel/region contains motion oran LED pulse. Additional details on step 850 are illustrated in FIG. 9.If the pixel in the corresponding region is determined to be an LED, at860 a control signal is output (for each determined pixel) that weightsthe pixel contribution to the final frame based on the LEDdetermination. Intensity data from a respective fusion or WDR imagereflecting this output would then be used in the blended image.(Generally, this would likely be the higher intensity contribution ofthe greater of the fusion image or the WDR image to that pixel/region inthe corresponding region of the blended image.) If the pixel in thecorresponding region is determined to be motion, at 870 a control signalis output (for each determined pixel) that weights the pixelcontribution to the final frame based on the motion determination. At880, the process continues for each pixel and region in the image.

FIG. 9 is a flowchart illustrating one embodiment of performing step850—determining whether an image includes motion or and LED pulse—inFIG. 8. At 900, the process takes as input the data from a long exposureimage (LONG), a short exposure image (SHORT) (adjacent in time to thelong exposure image), the fusion image data which may be an addition ofthe LONG data and the SHORT data or a color corrected output for theLONG and SHORT data, and the WDR image data (which is one of following:the LONG image data, the SHORT image data, or the blended of the LONGand SHORT image data). An LED's pulse frequency range is very wide and,LEDs in the field are likely to have many LEDs with differentfrequencies in an image. When there is an LED flickering, the differencebetween long exposure pixel value and short exposure pixel value is verylarge (LONG_(t)>>SHORT_(t)). For a moving object, the relationship ofprevious pair of LONG_(t-1) and SHORT_(t-1) will be(LONG_(t-1)≈SHORT_(t-1)). However, for a true LED flickering, theprevious pair of LONG_(t-1) and SHORT_(t-1) will still be(LONG_(t-1)>>SHORT_(t-1)).

One case is shown to determine whether an analysis needs to be performed(step 910) or whether, based on the data, an LED pulse can be determined(step 920 or 930).

An analysis is made at 940 if step 910 is true, or 920 and 930 arefalse. At 910, the LONG image data is much higher in intensity than theSHORT image data. In this case, the fusion image in this region willlikely contain a color corrected (blended) set of data and the WDR imagewill likely contain the SHORT image data (having likely determinedmotion in the raw data). If step 910 is true, the process moves to step940 where it is further determined if there is an LED pulse by comparingthe conditions of previous LONG and SHORT images to the current LONG andSHORT images. If there is an LED pulse, the conditions of these twopairs of LONG and SHORT images will be similar.

At 920, the SHORT image data is much higher in intensity than the LONGimage data. In this case, the fusion image in this region will likelycontain the SHORT data and the WDR image will likely contain the SHORTimage data (having likely determined no motion in the raw data). At 930,the LONG image data is close in intensity to the SHORT image data. Inthe case the fusion image in this region will likely contain the blendeddata and the image from the WDR will be blended data. If either 920 or930 are true, then the method determines that no LED is present in theregion.

The control signal output by the LED determination processor allows theblending processor to determine whether the WDR image data, or thefusion image data, or the blended of WDR and fusion images will be usedfor the output image. As described below, the blending processor choosesdata from one of the WDR image or the fusion image on a pixel by pixelbasis to generate the blended image. The blended image thus has anaccurate detection of LED lighting within the image than standard WDRimages while allowing the use of standard CMOS sensor technology.

FIG. 10 illustrates the process performed by the blending processor 450or blending module 520D. The blending processor 450 or blending module520D takes as input the fusion image at 1000 and the WDR image at 1010.For each pixel in each of the fusion image and the WDR image at 1020, at1030 and responsive to the control signal from the LED detectionprocessor, the blending processor 450 or blending module 520D selectsthe pixel value of the fusion image or the WDR image or a blendedversion of the two images for use in the blended image. The processcontinues for each pixel at 1070 until all pixels have been processed,at which point the process outputs the final blended image with improvedLED detection at 1080. As such, the blending processor or 450 orblending module 520D will output one of the following images: the WDRimage, the fusion image, or the blended of WDR and fusion images. If theLED detection processor 440 or module 520C can determine that a pixel isin LED flickering area, the blending processor 450 or blending module520D will output the fusion result for that pixel. On the other hand, ifthe LED detection processor 440 or module 520C can determine that apixel is in a motion area, the blending processor 450 or blending module520D will output the WDR result for that pixel. However, if a pixelcannot be certainly determined either in an LED flickering area or in amotion area, the blending processor 450 or blending module 520D willoutput the blended result of WDR and fusion images for that pixel. Theoperation of all modules is pixel based. To increase accuracy, thedetection operation will also reference the surrounding pixels of atarget pixel, it will be region based.

The technology may further include a means 400 for receiving sequentiallong and short exposure images output from an image sensor 410; a means430 for generating a wide dynamic range (WDR) image based on adjacentsequential long and short exposure images; a means 420 for generating afusion image based on the adjacent sequential long and short exposureimages, the fusion image comprising added data from each of the adjacentlong and short exposure images; a means 440 for generating a controlsignal based on the fusion image, the WDR image and the adjacentsequential long and short exposure images, the control signal indicatingwhether pixels in the fusion image or the WDR image comprise LED pulses;and a means 450 for generating a blended image responsive to the controlsignal by selecting data from one of the WDR image or the fusion image.

As noted above, the image processing technology discussed hereinimproves LED detection for a wide variety of applications. Thetechnology allows conventional CMOS image sensors to be used with a WDRprocessor or module output while mitigating LED flicker in a final,blended image. Hence, manufacturing costs can be relatively low. Inaddition, compared to the conventional LED capture and detectionprocesses, the technology can also improve the picture quality from theperspective of signal to noise ratio (SNR). The output of a WDRprocessor or module as discussed herein is a WDR image comprising acombination of LONG and SHORT exposure frames. The blending ratio forthis WDR image depends on the difference of LONG and SHORT exposuredata. For a pixel in a moving area, this difference will become largerand then the output pixel will include more SHORT exposure data. On theother hand, if a pixel is in a still and bright area, the differencewill be smaller and then the output pixel will include more LONGexposure data. In general, the SHORT exposure data has less motion blurbut is noisier than the LONG exposure data. When a detected pixel is ina still but darker area, the output of WDR module may include more SHORTexposure data because its corresponding SHORT exposure pixel is noisierand then the difference may become larger. In order to avoid suchinaccurate detection, the LED pulse detection processor 440 or module520C in can be used to check the relationship between the difference ofprevious pair (LONG_(t-1), SHORT_(t-1)) and the difference of currentpair (LONG_(t), SHORT_(t)). If these two differences are similar, thedetected pixel is in either LED flickering area or in still area, theblending module will output higher ratio of the fusion data in the finaldata. As mention above, the fusion image is the result of adding LONGand SHORT image data instead of only SHORT image data. Therefore, theSNR of the blended image will be better than the WDR image output in anoisy environment.

Disclosed herein is an image processing device comprising: a fusionprocessor configured to receive sequential long and short exposureimages and generate a fusion image comprising added correspondingportions of adjacent sequential long and short exposure images; a LEDdetection processor configured to receive the fusion image, thesequential long and short exposure images, and a wide dynamic range(WDR) image created from the sequential long and short exposure images,and generate a control signal indicating whether the correspondingregion includes LED flickers or motions; and a blending processorresponsive to the control signal to generate a blended image.

The image processing device may include the aforementioned imageprocessing device wherein the fusion processor is configured to outputthe fusion image by adding pixel values from the adjacent long and shortsequential images together.

The image processing device may include any of the aforementioned imageprocessing devices wherein the fusion processor adds pixel values on apixel-by-pixel basis in corresponding regions of the adjacent long andshort sequential images.

The image processing device may include any of the aforementioned imageprocessing devices wherein the fusion processor includes a saturationchecking mechanism operating on a pixel-by-pixel basis.

The image processing device may include any of the aforementioned imageprocessing devices wherein the blending processor selects a pixel valuefor a blended image from one of a corresponding pixel value in thefusion image, or the WDR image or a combination of the fusion image andthe WDR image based on the control signal.

The image processing device may include any of the aforementioned imageprocessing devices wherein the LED detection processor is configured todetermine whether at least a corresponding region of two adjacentsequential LONG and SHORT exposure images, the fusion image, and the WDRimage includes motion or an LED pulse based on the relationship betweenpixel values in the adjacent sequential LONG and SHORT images.

The image processing device may include any of the aforementioned imageprocessing devices wherein one of a WDR processor, the fusion processorand the blending processor determines whether a region is oversaturatedand corrects the oversaturated region.

The image processing device may include any of the aforementioned imageprocessing devices wherein the blending image comprises an image showingan LED during a long exposure image of an image capture device whichcaptures and outputs the sequential long and short sequential exposureimages.

Also disclosed is a computer-implemented method for processing images,comprising: receiving sequential long and short exposure images and awide dynamic range (WDR) image based on adjacent sequential long andshort exposure images; generating a fusion image based on the adjacentsequential long and short exposure images, the fusion image comprisingadded data from each of the adjacent long and short exposure images;generating a control signal based on the fusion image, the WDR image andthe adjacent sequential long and short exposure images, the controlsignal indicating whether pixels in the fusion image or the WDR imagecomprise an LED flicker; and generating a blended image responsive tothe control signal by selecting data from one of the following: WDRimage, or the fusion image, or both.

The computer-implemented method may include any of the aforementionedcomputer implemented methods wherein generating a fusion image comprisesadding pixel values from the adjacent long and short sequential imagestogether.

The computer-implemented method may include any of the aforementionedcomputer implemented methods wherein generating a fusion imagecomprising adding pixel values on a pixel-by-pixel basis incorresponding regions of the adjacent long and short sequential images.

The computer-implemented method may include any of the aforementionedcomputer implemented methods wherein generating a fusion image includeslimiting pixel saturation on a pixel-by-pixel basis.

The computer-implemented method may include any of the aforementionedcomputer implemented methods wherein generating a blended image includesselecting a pixel value for the blended image from one of acorresponding pixel value in the fusion image or the WDR image or acombination of the fusion image and the WDR image based on the controlsignal.

The computer-implemented method may include any of the aforementionedcomputer implemented methods which generates a control signal todetermine whether at least a corresponding region of two adjacentsequential long and short exposure images, the fusion image, and the WDRimage includes motion or an LED flickering based on the relationshipbetween pixel values in the adjacent sequential LONG and SHORT images.

The computer-implemented method may include any of the aforementionedcomputer implemented methods wherein generating a blended image includesdetermining whether a region in the WDR image, the fusion image or theblended image is oversaturated correcting the oversaturated region.

Also disclosed is a non-transitory computer-readable medium storingcomputer instructions for processing images, that when executed by oneor more processors, cause the one or more processors to perform thesteps of: receiving sequential long and short exposure images and a widedynamic range (WDR) image based on adjacent sequential long and shortexposure images; generating a fusion image based on the adjacentsequential long and short exposure images, the fusion image comprisingadded data from each of the adjacent long and short exposure images;generating a control signal based on the fusion image, the WDR image andthe adjacent sequential long and short exposure images, the controlsignal indicating whether pixels in the fusion image or the WDR imagecomprise LED flickering; and generating a blended image responsive tothe control signal by selecting data from one of the WDR image or thefusion image.

The non-transitory computer-readable medium may include any of theaforementioned non-transitory computer-readable mediums whereingenerating a fusion image comprises adding pixel values from theadjacent long and short sequential images together on a pixel-by-pixelbasis in corresponding regions of the adjacent long and short sequentialimages.

The non-transitory computer-readable medium may include any of theaforementioned non-transitory computer-readable mediums whereingenerating a fusion image includes limiting pixel saturation on apixel-by-pixel basis.

The non-transitory computer-readable medium may include any of theaforementioned non-transitory computer-readable mediums wherein thegenerating a control signal based on the fusion image determines whetherat least a corresponding region of two adjacent sequential long andshort exposure images, the fusion image, and the WDR image includesmotion or an LED on pulse based on the difference between the previouspairs of LONG and SHORT images and the current pairs of LONG and SHORTimages.

The non-transitory computer-readable medium may include any of theaforementioned non-transitory computer-readable mediums wherein thegenerating a blended image includes determining whether a region in theWDR image, the fusion image or the blended image is oversaturated andcorrecting the oversaturated region.

It is understood that the present subject matter may be embodied in manydifferent forms and should not be construed as being limited to theembodiments set forth herein. Rather, these embodiments are provided sothat this subject matter will be thorough and complete and will fullyconvey the disclosure to those skilled in the art. Indeed, the subjectmatter is intended to cover alternatives, modifications and equivalentsof these embodiments, which are included within the scope and spirit ofthe subject matter as defined by the appended claims. Furthermore, inthe following detailed description of the present subject matter,numerous specific details are set forth in order to provide a thoroughunderstanding of the present subject matter. However, it will be clearto those of ordinary skill in the art that the present subject mattermay be practiced without such specific details.

Aspects of the present disclosure are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatuses(systems) and computer program products according to embodiments of thedisclosure. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable instruction executionapparatus, create a mechanism for implementing the functions/actsspecified in the flowchart and/or block diagram block or blocks.

The computer-readable non-transitory media includes all types ofcomputer readable media, including magnetic storage media, opticalstorage media, and solid state storage media and specifically excludessignals. It should be understood that the software can be installed inand sold with the device. Alternatively the software can be obtained andloaded into the device, including obtaining the software via a discmedium or from any manner of network or distribution system, including,for example, from a server owned by the software creator or from aserver not owned but used by the software creator. The software can bestored on a server for distribution over the Internet, for example.

Computer-readable storage media (medium) exclude (excludes) propagatedsignals per se, can be accessed by a computer and/or processor(s), andinclude volatile and non-volatile internal and/or external media that isremovable and/or non-removable. For the computer, the various types ofstorage media accommodate the storage of data in any suitable digitalformat. It should be appreciated by those skilled in the art that othertypes of computer readable medium can be employed such as zip drives,solid state drives, magnetic tape, flash memory cards, flash drives,cartridges, and the like, for storing computer executable instructionsfor performing the novel methods (acts) of the disclosed architecture.

The terminology used herein is for the purpose of describing particularaspects only and is not intended to be limiting of the disclosure. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof.

The description of the present disclosure has been presented forpurposes of illustration and description, but is not intended to beexhaustive or limited to the disclosure in the form disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of thedisclosure. The aspects of the disclosure herein were chosen anddescribed in order to best explain the principles of the disclosure andthe practical application, and to enable others of ordinary skill in theart to understand the disclosure with various modifications as aresuited to the particular use contemplated.

For purposes of this document, each process associated with thedisclosed technology may be performed continuously and by one or morecomputing devices. Each step in a process may be performed by the sameor different computing devices as those used in other steps, and eachstep need not necessarily be performed by a single computing device.

Although the subject matter has been described in language specific tostructural features and/or methodological acts, it is to be understoodthat the subject matter defined in the appended claims is notnecessarily limited to the specific features or acts described above.Rather, the specific features and acts described above are disclosed asexample forms of implementing the claims.

1. An image processing device, comprising: a fusion processor configured to receive sequential long and short exposure images and generate a fusion image comprising added corresponding regions of adjacent sequential long and short exposure images; a LED detection processor configured to receive the fusion image, the sequential long and short exposure images and a wide dynamic range (WDR) image created from the sequential long and short exposure images, and generate a control signal indicating whether a corresponding region includes LED flickers or motion; and a blending processor responsive to the control signal to generate a blended image.
 2. The image processing device of claim 1 wherein the fusion processor is configured to output the fusion image by adding pixel values from the adjacent long and short sequential images together.
 3. The image processing device of claim 2 wherein the fusion processor adds pixel values on a pixel-by-pixel basis in corresponding regions of the adjacent long and short sequential images.
 4. The image processing device of claim 3 wherein the fusion processor includes a saturation checking mechanism operating on a pixel-by-pixel basis.
 5. The image processing device of claim 1 wherein the blending processor selects a pixel value for a blended image from one of a corresponding pixel value in the fusion image, the WDR image or a combination of the fusion image and the WDR image based on the control signal.
 6. The image processing device of claim 1 wherein the LED detection processor is configured to determine whether at least a corresponding region of two adjacent sequential long and short exposure images, the fusion image, and the WDR image includes motion or an LED pulse based on a relationship between pixel values in the adjacent sequential long and short images.
 7. The image processing device of claim 1 wherein one of a WDR processor, the fusion processor and the blending processor determines whether a region is oversaturated and corrects the oversaturated region.
 8. The image processing device of claim 1 wherein the blended image comprises an image showing an LED during a long exposure image of an image capture device which captures and outputs the sequential long and short exposure images.
 9. A computer-implemented method for processing images, comprising: receiving sequential long and short exposure images and a wide dynamic range (WDR) image based on adjacent sequential long and short exposure images; generating a fusion image based on the adjacent sequential long and short exposure images, the fusion image comprising added data from each of the adjacent long and short exposure images; generating a control signal based on the fusion image, the WDR image and the adjacent sequential long and short exposure images, the control signal indicating whether pixels in the fusion image or the WDR image comprise an LED flicker; and generating a blended image responsive to the control signal.
 10. The computer-implemented method of claim 9 wherein generating a fusion image comprises adding pixel values from the adjacent long and short sequential images together.
 11. The computer-implemented method of claim 10 wherein generating a fusion image comprising adding pixel values on a pixel-by-pixel basis in corresponding regions of the adjacent long and short sequential images.
 12. The computer-implemented method of claim 10 wherein generating a fusion image includes limiting pixel saturation on a pixel-by-pixel basis.
 13. The computer-implemented method of claim 9 wherein generating a blended image includes selecting a pixel value for the blended image from one of a corresponding pixel value in the fusion image, the WDR image, or a combination of the fusion image and the WDR image based on the control signal.
 14. The computer-implemented method of claim 9 wherein generating a control signal based on the fusion image determines whether at least a corresponding region of two adjacent sequential long and short exposure images, the fusion image, and the WDR image includes motion or an LED flicker based on a relationship between pixel values in the adjacent sequential long and short images.
 15. The computer-implemented method of claim 9 wherein generating a blended image includes determining whether a region in the WDR image, the fusion image or the blended image is oversaturated correcting any oversaturation.
 16. A non-transitory computer-readable medium storing computer instructions for processing images, that when executed by one or more processors, cause the one or more processors to perform the steps of: receiving sequential long and short exposure images and a wide dynamic range (WDR) image based on adjacent sequential long and short exposure images; generating a fusion image based on the adjacent sequential long and short exposure images, the fusion image comprising added data from each of the adjacent long and short exposure images; generating a control signal based on the fusion image, the WDR image and the adjacent sequential long and short exposure images, the control signal indicating whether pixels in the fusion image or the WDR image comprise an LED flicker; and generating a blended image responsive to the control signal by selecting data from one of the WDR image or the fusion image.
 17. The non-transitory computer-readable medium of claim 16 wherein generating a fusion image comprises adding pixel values from the adjacent long and short sequential images together on a pixel-by-pixel basis in corresponding regions of the adjacent long and short sequential images.
 18. The non-transitory computer-readable medium of claim 17 wherein generating a fusion image includes limiting pixel saturation on a pixel-by-pixel basis.
 19. The non-transitory computer-readable medium of claim 18 wherein the generating a control signal based on the fusion image determines whether at least a corresponding region of two adjacent sequential long and short exposure images, the fusion image, and the WDR image includes motion or an LED on pulse based on a relationship between pixel values the adjacent sequential long and short images.
 20. The non-transitory computer-readable medium of claim 16 wherein the generating a blended image includes determining whether a region in the WDR image, the fusion image or the blended images is oversaturated and correcting any oversaturation. 